1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a computer program that mix artificial images on On Screen Display (OSD) layers with main images on video layers and display the images resulting from the mixing. The main images are captured by image pickup devices, such as cameras, and are recorded or transmitted. The artificial images indicate information including the states of the apparatuses and the operating conditions. Particularly, the present invention relates to an image processing apparatus, an image processing method, and a computer program that are provided with memory areas only for the OSD, in addition to memory areas for video signals, and that mix the images on the OSD layers with the video signals for the OSD to display a variety of information on screens.
More specifically, the present invention relates to an image processing apparatus, an image processing method, and a computer program that perform the OSD in accordance with multiple image sizes corresponding to different resolutions of output images and different output formats of video signals. Particularly, the present invention relates to an image processing apparatus, an image processing method, and a computer program that have only one OSD layer having a reference size and that enlarge the image on the OSD layer in accordance with the image size of a video signal to be output to realize the OSD.
2. Description of the Related Art
Video cameras, digital cameras, etc. are in widespread use in recent years. Such video devices are generally provided with monitor displays displaying main images that are captured by image pickup devices, such as cameras, and are recorded or transmitted, and information indicating various settings is also displayed on the monitor displays. The main images are natural images composed of video signals captured by the cameras while the information about the various settings is artificial images, such as graphics, characters, or texts, generated by computers or the likes. The information about the various settings includes a recording or playback time, a recording-playback mode, and the remaining amount of battery. Such information is often superimposed on the main images in an “OSD” format.
The “OSD” is a very important function for displaying the states and/or the operating conditions of an apparatus and indicating to users how to operate the apparatus while main images are displayed. Most video devices currently manufactured and sold have the OSD function.
Digital cameras are commonly used as media for image information. For example, Japanese Unexamined Patent Application Publication No. 11-146234 discloses a digital camera that is capable of operating in a mode in which image data stored in a memory card is displayed or played back and in a mode in which a captured image is directly displayed. The digital camera mixes image data with character data indicating a counter, a date, image capturing conditions, etc. on the screen of a liquid crystal display (LCD) panel, which is a display apparatus, to realize the OSD.
For example, Japanese Unexamined Patent Application Publication No. 7-162750 discloses an on-screen control apparatus including a unit that scales a character or pattern into an arbitrary size, a unit that displays the character or pattern at an arbitrary position, and an operation unit on which a key used for scaling the character or pattern into an arbitrary size and a key used for moving the character or pattern to an arbitrary position are provided in order to display an on-screen image including the character or pattern on a monitor in a relatively east-to-see size and at a relatively easy-to-see position. The on-screen control apparatus displays the character or pattern in an arbitrary size and at an arbitrary position in response to operations with the keys.
Enlargement of the display sizes of icons involved in user settings allows the visibility of on-screen images to be improved. For example, Japanese Unexamined Patent Application Publication No. 2005-301992 discloses an electronic apparatus with a display, which determines a change in the state of the apparatus or in the operation condition that is set, identifies the icon indicating the changed state of the apparatus or the changed operation condition, and performs the OSD in a mode different from that of the previous icon.
Initial OSD only displays simple characters or graphics in one color and can be performed only by switching between a video signal and an OSD signal at vertical horizontal addresses specified on the display. However, information display during recording and playback becomes important along with an increasing range of functions of recent devices, and the OSD is necessary to achieve higher quality, higher function, and an increased number of colors. In addition, the design of the OSD is also emphasized for differentiation of products and it is necessary to display the OSD with higher quality as in natural images having higher resolution and multi-valued gradation.
Accordingly, methods are widely adopted at present, in which an OSD area is provided in part of a large scale memory where images are temporarily stored (or a memory area only for the OSD is provided, in addition to a memory area for video signals) and an OSD layer read out from the corresponding memory area is mixed with an image layer read out from the corresponding memory area at a constant ratio to realize the OSD with higher quality.
Many video devices including video cameras and digital cameras perform line output. Video signals in the line output have various formats, such as analog National Television System Committee (NTSC), analog Phase Alternating Line (PAL) (Standard Definition (SD)), and HiVision, and also have derived specifications for various image sizes, such as D1, D2, D3, and D4. Preparing the OSD for the respective formats and regenerating OSD layers each time the format is switched impose heavy loads on firmware and program read only memories (ROMs).
Some video cameras and digital cameras are capable of concurrently performing panel display and line output. The panel display normally has image sizes (resolutions) including Video Graphics Array (VGA) (horizontal 640 pixels×vertical 480 lines) and Super VGA (SVGA) (horizontal 800 pixels×vertical 600 lines), and OSD layers can be generated in accordance with the image sizes and can be mixed with main images to realize the OSD with higher quality. In contrast, since the line output has image sizes different from those of the panel display, such as the D1 (horizontal 720 pixels×vertical 480 lines) and the D3 (horizontal 1,920 pixels×vertical 1,080 lines), it is necessary to prepare the OSD layers for the line output separately from the OSD layers for the panel display. However, it is necessary to generate the OSD layers of the respective image sizes in the OSD area on a large scale random access memory (RAM) in order to concurrently output the OSD layers of the image sizes appropriate for the various output formats and, therefore, the OSD layers consume a significant capacity of the large scale RAM.
In general, in order to perform the OSD in accordance with the multiple specifications for the image sizes, only one OSD layer having a reference size is prepared and the OSD layer is appropriately enlarged and output in accordance with the image size when the mode of the image size of a video signal to be output is switched. The reference size of the OSD layer specifically means a minimum size that is normally involved and can be appropriately enlarged in accordance with the image size of an output destination.
Two methods are used to enlarge the OSD layer having the reference size: one method uses a filter to enlarge the OSD layer and the other method reads out a pixel or line twice on a predetermined cycle to enlarge the OSD layer.
In the former enlargement method, natural images can be enlarged or reduced by using an interpolation filter based on a sampling theorem, such as bilinear interpolation or Cubic interpolation. Accordingly, it is possible to generate enlarged images with higher quality in the case of the natural images having the multi-valued gradation and including noises in principle. However, since enlargement of artificial images by using the above interpolation filter rounds edges of, for example, characters, it is not possible to generate enlarged images with higher quality. Particularly, binary images, such as characters or texts, are often enlarged by nearest neighbor interpolation in order to avoid blurring caused by the rounded edges. However, the enlargement process by the nearest neighbor interpolation has a visual problem of jaggy (the jaggy observed at the outlines of images).
Accordingly, the latter method of reading out a pixel or line twice on a predetermined cycle to enlarge the OSD layer will now be considered.
For example, Japanese Unexamined Patent Application Publication No. 11-338454 discloses an on-screen display circuit having a decreased memory size to realize the OSD function. The on-screen display circuit realizes the decrease in the memory size by including a unit that repetitively reads out data on the same line of the bitmap in n lines in the OSD from a storage unit to vertically enlarge the OSD at a scaling factor “n” in vertical n-times enlargement and that reads out the lines of the bitmap for every n lines to vertically reduce the OSD at a scaling factor “1/n” in vertical 1/n-times reduction.
For example, Japanese Unexamined Patent Application Publication No. 2005-99516 discloses an image display apparatus that newly generates a horizontal synchronization signal for external synchronization by a synchronization signal generating circuit and supplies the generated horizontal synchronization signal to an external synchronization input of an external graphic generating apparatus, when a first graphic image supplied from the external graphic generating apparatus is converted into a second graphic image having effective lines of a number different from that of the first graphic image, to match the line display speed of the first graphic image output from the external graphic generating apparatus with the vertical size of the effective image area of a display apparatus (or a display unit) where the second graphic image is displayed.
When the image size on the OSD layer is enlarged at a scaling factor of an integer, for example, at a scaling factor of two or three, the same pixel is constantly read out twice or three times horizontally and vertically to generate a natural enlarged character or graphic. However, when the ratio between the image size of an original OSD layer and the image size of an output layer is not an integer, it is necessary to enlarge the horizontal and vertical image sizes of the OSD layer at a scaling factor of a real number. In the method of reading out an image twice to enlarge the image size at a scaling factor of a real number, the same pixel is read out twice only for every several pixels. In such a case, when a horizontal or vertical line written by one pixel arrives at the position where the double reading is performed, the horizontal or vertical line is doubled in thickness and loses the matching with other parts. In addition, when the same pixel is read out twice for every several pixels, enlargement of an oblique line results in a jaggy line, as shown in FIG. 10, and the design of characters or graphics can be damaged.
Output of an OSD layer generated for the panel display in an irrelevant image-size format, such as a line output format, will now be described.
For example, when the panel attached to the main body of a video device has the VGA size (horizontal 640 pixels×vertical 480 lines), the OSD layer is often generated in the same size (horizontal 640 pixels×vertical 480 lines). However, when the OSD layer is output in the analog NTSC line output format, it is necessary to perform the enlargement from 640 pixels to 720 pixels because the number of horizontal pixels in the NTSC is equal to 720 pixels. Since the enlargement from 640 pixels to 720 pixels is basically performed by repeating the enlargement process from eight pixels to nine pixels, the same pixel is repeated every eight pixels, as shown in FIG. 11.
When the analog PAL format is selected as the line output format of an OSD layer having the VGA size, it is necessary to output 288 lines in one field because the number of vertical lines in the PAL format is equal to 576 and the PAL format adopts an interlace method. Accordingly, as shown in FIG. 12, an original OSD layer is read out every other line and the next line is read out twice for every six lines to perform the enlargement of the OSD layer.
When the analog 3D format (horizontal 1,440 pixels×vertical 1,080 pixels) is selected as the line output format of an OSD layer having the VGA size, it is necessary to horizontally enlarge the 640 pixels to 1,440 pixels. Accordingly, as shown in FIG. 13, the same pixel is read out twice and the readout of the same pixel is repeated three times for every four pixels. In addition, since 540 lines should be output in one field because the interlace scan is normally performed, it is necessary to vertically enlarge 480 lines to 540 liens. In this case, the process of enlarging eight lines to nine lines is repeated and the same line is read out twice for every eight liens.
The image-size format of the panel or electronic view finder (EVF) display has no relation to the image size of the line output. Accordingly, in order to fit the OSD layer that is generated so as to be fitted in the image size of the panel or EVF display in the image size of the line output, it is necessary to perform the above process of performing the double reading of a pixel every several pixels. Such a conversion process of the image size is not in an irregular conversion mode that occurs at a lower frequency at which the damage to characters or graphics can be permitted but is in a normal mode that occurs at a considerably higher frequency. In other words, it is not possible to ignore such a conversion process.